Containerized workload scheduling

ABSTRACT

A method for containerized workload scheduling can include determining a network state for a first hypervisor in a virtual computing cluster (VCC). The method can further include determining a network state for a second hypervisor. Containerized workload scheduling can further include deploying a container to run a containerized workload on a virtual computing instance (VCI) deployed on the first hypervisor or the second hypervisor based, at least in part, on the determined network state for the first hypervisor and the second hypervisor.

BACKGROUND

Virtual computing instances (VCIs), such as virtual machines, virtualworkloads, data compute nodes, clusters, and containers, among others,have been introduced to lower data center capital investment infacilities and operational expenses and reduce energy consumption. A VCIis a software implementation of a computer that executes applicationsoftware analogously to a physical computer. VCIs have the advantage ofnot being bound to physical resources, which allows VCIs to be movedaround and scaled to meet changing demands of an enterprise withoutaffecting the use of the enterprise's applications. VCIs can be deployedon a hypervisor provisioned with a pool of computing resources (e.g.,processing resources, memory resources, etc.). There are currently anumber of different configuration profiles for hypervisors on which VCIsmay be deployed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a host for microservice scheduling according tothe present disclosure.

FIG. 2 is a diagram of a simplified system for microservice schedulingaccording to the present disclosure.

FIG. 3A is a diagram of a system including a scheduling agent, virtualcomputing instances, and hypervisors for microservice schedulingaccording to the present disclosure.

FIG. 3B is another diagram of a system including a scheduling agent,virtual computing instances, and hypervisors for microservice schedulingaccording to the present disclosure.

FIG. 4A is a flow diagram representing a method for microservicescheduling according to the present disclosure.

FIG. 4B is another flow diagram representing a method for microservicescheduling according to the present disclosure.

FIG. 5 is a diagram of a system for microservice scheduling according tothe present disclosure.

FIG. 6 is a diagram of a machine for microservice scheduling accordingto the present disclosure.

DETAILED DESCRIPTION

The term “virtual computing instance” (VCI) covers a range of computingfunctionality. VCIs may include data compute nodes such as virtualmachines (VMs). Containers, also referred to herein as a “containerizedworkload,” can run on a host operating system without a hypervisor orseparate operating system, such as a container that runs within Linux. Acontainer can be provided by a virtual machine that includes a containervirtualization layer (e.g., Docker). A VM refers generally to anisolated end user space instance, which can be executed within avirtualized environment. Other technologies aside from hardwarevirtualization can provide isolated end user space instances may also bereferred to as VCIs. The term “VCI” covers these examples andcombinations of different types of VCIs, among others.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. The hostoperating system can use name spaces to isolate the containers from eachother and therefore can provide operating-system level segregation ofthe different groups of applications that operate within differentcontainers. This segregation is akin to the VM segregation that may beoffered in hypervisor-virtualized environments that virtualize systemhardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers may be more “lightweight” than VMs at leastbecause they share an operating system rather than operating with theirown guest operating system.

Multiple VCIs can be configured to be in communication with each otherin a software defined data center. In such a system, information can bepropagated from an end user to at least one of the VCIs in the system,between VCIs in the system, and/or between at least one of the VCIs inthe system and a non-virtualized physical host.

Software defined data centers are dynamic in nature. For example, VCIsand/or various application services, may be created, used, moved, ordestroyed within the software defined data center. When VCIs are created(e.g., when a container is initialized), various processes and/orservices start running and consuming resources. As used herein,“resources” are physical or virtual components that have a finiteavailability within a computer or software defined data center. Forexample, resources include processing resources, memory resources,electrical power, and/or input/output resources, etc.

Containerized cloud-native applications can be used to accelerateapplication delivery in software defined data centers. As used herein,“containerized” or “containerization” refers to a virtualizationtechnique in which an application (or portions of an application, suchas flows corresponding to the application) are encapsulated into acontainer (e.g., Docker, Linux containers, etc.) as an alternative tofull machine virtualization. Because containerization can includeloading the application on to a VCI, the application may be run on anysuitable physical machine without worrying about applicationdependencies. Further, as used herein, “cloud-native applications” referto applications (e.g., computer programs, software packages, etc.) thatare assembled as microservices in containers deployed in a softwaredefined data center. “Microservices” refer to a computing architecturein which an application is structured as a collection of loosely coupled(e.g., containerized) services. Microservice architectures may allow forimproved application modularity, scalability, and continuous deploymentin comparison to traditional application development environments.

In order to take advantage of the perceived benefits of containerizedcloud-native applications, container schedulers such as KUBERNETES®,DOCKER SWARM®, MESOS®, etc. can be used to deploy and/or managecontainerized applications. Container schedulers can consider parametersassociated with the software defined data center on which they operateto deploy and/or manage the containerized applications. In someapproaches, the parameters considered by the container scheduler caninclude host VCI resources (e.g., host VCI processing resources and/ormemory resources), host VCI processing resource and/or memory resourceutilization, and/or policy-based affinity rules (e.g., policy-basedrules that can control the placement of VCIs and/or containers on hostmachines within a virtual cluster) as part of scheduling deploymentand/or managing containers. This may be sub-optimal as the requirementsof software defined data centers continue to expand.

For example, software defined data centers currently host a widespectrum of applications with different needs, and therefore disparateapplication performance requirements. As the use of software defineddata centers continues to increase, the spectrum of applications hostedon software defined data centers will continue to increase, furtheremphasizing the disparate performance requirements of the applications.For example, due to the dynamic nature of applications deployed in asoftware defined data center (e.g., applications running on VCIs,computers, etc. of the software defined data center), resourcerequirements of the applications may evolve over time, which can leadsituations in which some approaches fail to adequately address evolvingapplication performance requirements.

In order to address the dynamic nature of applications hosted onsoftware defined data centers, embodiments disclosed herein can allowfor a container scheduler to consider network states for the softwaredefined data center such as VCI network states, hypervisor networkstatistics, application latencies (e.g., latency requirements ofmicroservices deployed on containers), and network overlay parameters,among others described in more detail herein, when scheduling containersand/or microservices. For example, a method for microservice schedulingcan include determining a network state for a first hypervisor in avirtual computing cluster (VCC). The method can further includedetermining a network state for a second hypervisor. Microservicescheduling can further include deploying a container to run amicroservice on a virtual computing instance (VCI) deployed on the firsthypervisor or the second hypervisor based, at least in part, on thedetermined network state for the first hypervisor and the secondhypervisor.

Other embodiments can include determining a network state for a firstVCI and a second VCI and deploying a container to run a microservice onthe first VCI or the second VCI based, at least in part, on thedetermined network state for the first VCI and the second VCI. Yet otherembodiments can include determining network state information for acluster (e.g., a virtual computing cluster) and scheduling containerand/or microservice deployment based on the network state information.

As used herein, the singular forms “a”, “an”, and “the” include singularand plural referents unless the content clearly dictates otherwise.Furthermore, the words “can” and “may” are used throughout thisapplication in a permissive sense (i.e., having the potential to, beingable to), not in a mandatory sense (i.e., must). The term “include,” andderivations thereof, mean “including, but not limited to.”

The figures herein follow a numbering convention in which the firstdigit or digits correspond to the drawing figure number and theremaining digits identify an element or component in the drawing.Similar elements or components between different figures may beidentified by the use of similar digits. For example, 106 may referenceelement “06” in FIG. 1 , and a similar element may be referenced as 206in FIG. 2 . A group or plurality of similar elements or components maygenerally be referred to herein with a single element number. Forexample a plurality of reference elements 106-1, 106-2, . . . , 106-Nmay be referred to generally as 106. As will be appreciated, elementsshown in the various embodiments herein can be added, exchanged, and/oreliminated so as to provide a number of additional embodiments of thepresent disclosure. In addition, as will be appreciated, the proportionand the relative scale of the elements provided in the figures areintended to illustrate certain embodiments and should not be taken in alimiting sense.

Embodiments of the present disclosure are directed to microservicescheduling, for example, in the context of a software defined datacenter (e.g., a distributed computing environment) including one or moreVCIs and/or containers. As described above, “microservices” refer tocontainerized instructions that correspond to portions of an applicationand are structured as a collection of loosely coupled (e.g.,containerized) services. Microservices can be created using differentcoding languages (e.g., as part of a polyglot approach to applicationdeployment). For example, in a microservices architecture, anapplication can be divided into multiple modular services that can bedeployed on containers. The microservices can run fine grained services,and the containers can have short lifespans. As used herein,“fine-grained services” refer to services that make direct use ofresources that are granted direct access by one or more applicationprogramming interfaces (APIs). In contrast, coarse grained servicesinclude services that utilize multiple fine-grained services. Further,as used herein, a “short lifespan” refers to a container that isdestroyed after a short period of time (e.g., seconds to minutes), ascompared to “long lifespan” containers, which operate for minutes ormore before being destroyed. In some embodiments, short lifespancontainers are containers that run microservices, which are generallydestroyed after a relatively short period of time once the microservicehas been executed and consumed by an application.

Due to the short-lived nature of containers on which microservices aredeployed, haphazard scheduling of the microservices can incur unwantedlatencies in application execution. For example, latencies associatedwith application execution can exceed desirable thresholds, which canreduce the efficacy of a software defined data center. In addition,network latencies and/or throughput between individual microservices canaffect performance of an application that is associated with themicroservices.

Embodiments herein may allow for improved scheduling of microserviceswhich can lead to improved performance of a computing system such as asoftware defined data center, virtual computing cluster, server, orother computing device. For example, by scheduling microservices inaccordance with the embodiments described herein, applications can beassembled from microservices more efficiently than in some approaches,which can reduce an amount of computing resources and/or an amount oftime required to execute the application. This can lead to reduceddowntime, quicker application execution, and/or improved userexperience.

FIG. 1 is a diagram of a host 102 for microservice scheduling accordingto the present disclosure. The system can include a host 102 withprocessing resource(s) 108 (e.g., one or more processors), memoryresource(s) 110 (e.g., one or more main memory devices and/or storagememory devices), and/or a network interface 112. The host 102 can beincluded in a software defined data center. A software defined datacenter can extend virtualization concepts such as abstraction, pooling,and automation to data center resources and services to provideinformation technology as a service (ITaaS). In a software defined datacenter, infrastructure, such as networking, processing, and security,can be virtualized and delivered as a service. A software defined datacenter can include software defined networking and/or software definedstorage. In some embodiments, components of a software defined datacenter can be provisioned, operated, and/or managed through anapplication programming interface (API).

The host 102 can incorporate a hypervisor 104 that can execute a numberof VCIs 106-1, 106-2, . . . , 106-N (referred to generally herein as“VCIs 106”). The VCIs can be provisioned with processing resources 108and/or memory resources 110 and can communicate via the networkinterface 112. The processing resources 108 and the memory resources 110provisioned to the VCIs 106 can be local and/or remote to the host 102(e.g., the VCIs 106 can be ultimately executed by hardware that may notbe physically tied to the VCIs 106). For example, in a software defineddata center, the VCIs 106 can be provisioned with resources that aregenerally available to the software defined data center and are not tiedto any particular hardware device. By way of example, the memoryresources 110 can include volatile and/or non-volatile memory availableto the VCIs 106. The VCIs 106 can be moved to different hosts (notspecifically illustrated), such that a different hypervisor manages theVCIs 106. In some embodiments, the host 102 can be connected to (e.g.,in communication with) a microservice scheduling agent 114, which can bedeployed on a VCI 106.

The VCIs 106-1, . . . , 106-N can include one or more containers (e.g.,containers 220 illustrated in FIG. 2 , herein), which can have amicroservice (e.g., microservice 222 illustrated in FIG. 2 , herein)running thereon. The microservices can correspond to one or moreapplications or portions of applications executed by the VCIs 106 and/orthe host 102. The application may be configured to perform certain tasksand/or functions for the VCIs 106 and/or the host 102. By executing theapplication using multiple microservices, scalability and/or portabilityof applications may be improved in comparison to approaches in whichapplications are monolithic.

The microservice scheduling agent 114 can be configured to causecontainer and/or microservice deployment across the VCIs 106, asdescribed in more detail, herein. In some embodiments, the microservicescheduling agent 114 can be deployed on (e.g., may be running on) thehost 102, and/or one or more of the VCIs 106.

In some embodiments, the microservice scheduling agent 114 can include acombination of software and hardware, or the microservice schedulingagent 114 can include software and can be provisioned by processingresource 108. An example of the microservice scheduling agent 114 isillustrated and described in more detail with respect to FIGS. 2 and 3 ,herein. Non-limiting examples of microservice scheduling agents 114 caninclude container schedulers such as KUBERNETES®, DOCKER SWARM®, MESOS®,etc.

FIG. 2 is a diagram of a simplified system for microservice schedulingaccording to a number of embodiments of the present disclosure. Thesystem 200 can include a pool of computing resources 216, a plurality ofVCIs 206-1, 206-2, . . . , 206-N, a microservice scheduling agent 207,and/or a hypervisor 204. As used herein, an “agent” is a computingcomponent configured to run at least one piece of software that isconfigured to perform actions without additional outside instruction.For example, an agent can be configured to execute instructions usingcomputing resources, such as hardware, that can be available to theagent in the pool of computing resources.

The system 200 can include additional or fewer components thanillustrated to perform the various functions described herein. In someembodiments, the VCIs 206-1, 206-2, . . . , 206-N, and/or themicroservice scheduling agent 207 can be deployed on the hypervisor 204and can be provisioned with the pool of computing resources 216;however, embodiments are not so limited and, in some embodiments, themicroservice scheduling agent 207 can be deployed on a VCI (e.g., amaster VCI or master node) that controls operation of the VCIs 206. Thislatter embodiment is described in more detail in connection with FIGS.3A and 3B, herein.

The pool of computing resources 216 can include physical computingresources used in a software defined data center, for example, compute,storage, and network physical resources such as processors, memory, andnetwork appliances. The VCIs 206-1, 206-2, . . . , 206-N, can beprovisioned with computing resources to enable functionality of the VCIs206-1, 206-2, . . . , 206-N. In some embodiments, the system 200 caninclude a combination of hardware and program instructions that areconfigured to provision the VCIs 206-1, 206-2, . . . , 206-N using apool of computing resources in a software defined data center.

In some embodiments, the microservice scheduling agent 207 can assignVCIs 206 to host the containers 220. For example, when a new container220 is generated to, for example, run a microservice 222, themicroservice scheduling agent 207 can select a VCI (e.g., VCI 206-1) onwhich to deploy the container (e.g., the container 220-1). As part ofselecting the VCI 206 on which to deploy the container 222, themicroservice scheduling agent 207 can receive information fromrespective scheduling sub-agents (e.g., the scheduling sub-agents 326-1,. . . , 326-N illustrated in FIGS. 3A and 3B, herein) corresponding toavailable resources of the VCIs 206. For example, the microservicescheduling agent 207 can receive information corresponding to an amountof processing resources and/or memory resources allocated to therespective VCIs 206, an amount of processing resources and/or memoryresources available to the respective VCIs 206, and/or network stateinformation related to the availability of the VCI 206 to run acontainer 220.

FIG. 3A is a diagram of a system including a scheduling agent 307,virtual computing instances 306, and hypervisors 304 for microservicescheduling according to the present disclosure. As shown in FIG. 3A, thesystem includes a microservice scheduling agent 307, a plurality of VCIs306-1, . . . , 306-N, and a plurality of hypervisors 304-1, . . . ,304-N. The plurality of VCIs 306 can include respective containers 320,which can run respective microservices 322 (e.g., microservices 322_(A)-1, . . . , 322 _(A)-N, 322 _(B), 322 _(M), etc.). In addition, therespective VCIs 306 can include respective scheduling sub-agents 326-1,. . . , 326-N. Non-limiting examples of scheduling sub-agents 326 caninclude KUBELETS®, among other scheduling sub-agents, that may bedeployed on the VCIs 306 to communicate resource information and/ornetwork state information corresponding to the VCIs 306 and/orhypervisors 304 on which they are deployed to the microservicescheduling agent 307. The VCIs 306 and hypervisors 304 illustrated inFIGS. 3A and 3B can, in some embodiments, be part of a cluster 305(e.g., a virtual computing cluster (VCC)).

Although the VCI 306-1 is shown as having a plurality of containersdeployed thereon (e.g., the containers 320 _(A)-1, . . . , 320 _(A)-N)and the other VCIs 306-2 and 306-N are illustrated as having a singlecontainer deployed thereon (e.g., the containers 320 _(B) and 320 _(M)),embodiments are not so limited and the VCIs 306 can include a greater orlesser number of containers based on the resources available to therespective VCIs 306.

The VCIs can include configuration agents 327-1, . . . , 327-N, such asa network virtualization platform node agent that can be configured toretrieve configuration (e.g., internet protocol, media access control,virtual local area network, etc.) information for the VCIs 306. Asdescribed in more detail below, in some embodiments, the configurationagents 327-1, . . . , 327-N can receive network state information fromthe data collection agents 328-1, . . . , 328-N deployed on thehypervisors 304-1, . . . , 304-N. Although shown in FIG. 3A as runningon the hypervisors 304, it will be appreciated that the data collectionagents 328 may run a bare metal server instead of on a hypervisor 304.

The containers 320 can be configured to run microservices 322 as part ofproviding an application to be executed by the microservice schedulingagent 307 and/or the VCIs 306. As described above, microservices caninclude instructions corresponding to modularized, containerizedportions of an application. Containers 320 that are runningmicroservices 322 can be “short lived” due to the nature of themicroservices. For example, the containers 320 that are runningmicroservices 322 may only be in existence for a short period (e.g.,seconds to minutes) of time, and may be destroyed after the microservice322 running thereon is no longer useful or needed. In some embodiments,the containers 320 that are running microservices 322 may be destroyedafter the microservice 322 running thereon has been executed and/or theapplication that was using the microservice 322 has been executed.

As a result, the microservices 322 can, in some embodiments, affectoverall system latency if execution of the microservices 322 is notscheduled effectively. In some approaches, microservices 322 may bescheduled (e.g., by the microservice scheduling agent 307) based solelyon resource consumption associated with the VCIs 306 on which thecontainers 320 to run the microservices 322 are deployed, however, byonly taking the resource consumption of the VCIs 306 into account whenscheduling execution of the microservices 322, other network parametersthat can affect the latency of the microservices 322 (or the applicationthat depends on the microservices) may not be taken into account whenscheduling execution of the microservices, which can result in degradedsystem and/or application performance.

In addition, system latency may be affected by processor (e.g., centralprocessing unit (CPU)) utilization of the hypervisors 304. For example,hypervisor 304 threads can perform additional network processing onbehalf of the VCIs 306, however, in some approaches these additional CPUcycles may not be retrievable from CPU statistics collected solely fromVCIs 306. As an example, if a VIC (e.g., the VCI 306-1) is operatingwith one virtual central processing unit (VCPU) and the CPU utilizationcollected from the VCI 306-1 is 60%, it is still possible that thenetwork processing being performed by the hypervisor (e.g., thehypervisor 304-1) on behalf of the VCI 306-1 is greater than the 60%collected from the VCI 306-1. For example, even if the CPU utilizationcollected from the VCI 306-1 is 60%, it is possible that the networkprocessing being performed by the hypervisor 304-1 on behalf of the VCI306-1 is nearly 100%. This can cause workloads running on the VCI 306-1to experience poor network performance since the hypervisor 304-1threads may not have enough CPU to perform additional network processingfor the VCI 306-1. However, in some embodiments, the data collectionagents 328 may have access to statistics regarding the hypervisor 304-1network processing loads, which can be utilized for microservicescheduling as described herein.

The hypervisors 304-1, . . . , 304-N can include data collection agents328-1, . . . , 328-N and interfaces 329-1, . . . , 329-N. The datacollection agents 328 can periodically or continually collectinformation such as statistics corresponding to various network statesof the cluster 305. For example, the data collection agents 328 cancollect statistics on the interfaces 329, which may be physical networkinterface controllers (pnics), overlay network information (e.g., tunnelstates), and/or network flows for the VCIs 306 that host containers 320.

Overlay network information can include information corresponding tonetwork traffic paths between the VCIs 306, hypervisors 304, themicroservice scheduling agent 307, and/or other components of thecluster 305. As the use of overlay networks can involve complicatedtraffic routing, performance of the software defined data center andapplications running thereon may be affected by the routing of trafficthrough the overlay network. To permit routing of traffic through theoverlay network, tunnels (e.g., secure shell tunnels) may be used toroute traffic between the hypervisors 304 and the VCIs 306 (e.g., to aport of the VCIs 306). By controlling the microservice scheduling agent307 to analyze and determine traffic routes through the cluster 305(e.g., by analyzing tunnel state information), the microservicescheduling agent 307 can provide improved microservice scheduling incomparison to approaches in which network states are not consideredduring microservice scheduling.

In some embodiments, the microservice scheduling agent 307 can collectand analyze statistics corresponding to the interfaces 329 as part of anoperation to schedule microservices (e.g., as part of an operation toschedule deployment of containers 320 on which microservices 322 arerun). For example, the microservice scheduling agent 307 can determinewhether flows in the cluster 305 are short lived (e.g., correspond tomicroservices and running on containers that exist for second tominutes) or are “long lived,” (e.g., high volume flows running oncontainers that exist for minutes or longer). The determination can bebased on a byte count and/or a time threshold associated with executionof a microservice 322 or application. For example, flows that exceed acertain quantity of bytes can be classified as long lived, while flowsthat do not exceed the certain quantity of bytes can be classified asshort lived. In the alternative or in addition, containers 322 that arein existence for seconds to minutes can be classified as short lived,while containers that are in existence for minutes or longer can beclassified as long lived.

In addition to, or in the alternative, the data collection agents 328can collect statistics corresponding to interference from non-containerVCIs co-located on hypervisors 304 where VCIs 306 are running acontainer 320. For example, in public cloud deployments, the datacollection agent(s) 328 can detect interference from non-containerizedresources that may be consuming VCI 306 resources that the microservicescheduling agent 307 may not be able to detect. In some embodiments,non-container VCIs are VCIs that do not have any containers deployedthereon and are instead running traditional workloads. By using thisinformation, microservice scheduling may be improved in comparison toapproaches in which a microservice scheduling agent 307 is unable todetect interference from non-containerized resources running on the VCIs306. Non-containerized workloads can include traditional workloads suchas public cloud, hypervisor deployed workloads and/or VCIs deployed onshared hypervisors.

The microservice scheduling agent 307 can use the determined information(e.g., the byte counts, time thresholds, or other network statesdescribed above) to generate scores for the VCIs 306. In someembodiments, the scores can fall within a range of 0-10 (with 10 beingthe best score and zero being the worst score), however, embodiments arenot limited to this particular example and the scores can be based onany scoring system.

In a non-limiting example in which the microservice scheduling agent 307is preparing to facilitate the scheduling of a new container 320 to runa microservice 322, the microservice scheduling agent 307 can assign ascore of 0 (zero) to a VCI (e.g., VCI 306-1) if the microservicescheduling agent 307 determines that a tunnel state corresponding to theVCI 306-1 is in a degraded state. In this example, the microservicescheduling agent 307 can assign the score of zero to the VCI 306-1because it may not yet be known if a container 320 to be deployed on oneof the VCIs 306 will require the use of overlay networking. If thehypervisors 304 are utilizing multiple uplinks, VCI capacity can beassumed to reflect a worst-case scenario in order to provide higherquality microservice scheduling. In some embodiments, the score can bebased on additional information described above (e.g., flow statistics,pnic statistics, etc.).

In another non-limiting example, if the VCI 306-2 is running ten longlived or heavy volume flows on the container 320 _(B) and another VCI(e.g., the VCI 306-N) is running two long lived heavy volume flows onthe containers 320 _(M) (noting that both the VCI 306-2 and the VCI306-N are running on the same hypervisor 304-N), the microservicescheduling agent 307 can assign a lower score to the VCI 306-2 than tothe VCI 306-N. Once the scores have been assigned, the microservicescheduling agent 307 can cause the score information to be sent to theconfiguration agents 327-2 and 327-N running on the VCIs 306-2 and306-N.

The configuration agents 327-1, . . . , 327-N can run on the VCIs 306and can communicate with the hypervisors 304 to receive configurationinformation from the data collection agents 328. The informationreceived form the data collection agents 328 can include informationrelevant to the VCI 306 on which the configuration agent 328 is running(e.g., the configuration agent 327-1 can receive informationcorresponding to the VCI 306-1, etc.). In some embodiments, theconfiguration agents 327 can forward this information on to themicroservice scheduling agent 307 to facilitate microservice scheduling,as described herein. For example, the configuration agents 327 canforward information corresponding to network states for the VCIs 306and/or the hypervisors 304 to the microservice scheduling agent 307 tobe used in the process of scheduling container deployment on the VCIS306 and microservice deployment on the containers 320.

A further non-limiting example of microservice scheduling in accordancewith the disclosure follows: When a cluster 305 is generated, the datacollection agents 328-1, . . . , 328-N on the hypervisors 304-1, . . . ,304-N can periodically (or continually) collect statistics on thenetwork states, as described above. While the data collection agents 328are collecting statistics on the network states, the data collectionagents 328 can generate scores for the VCIs 306 based on the networkstates as described above.

The data collection agents 328 can forward the information and/or thescores to the configuration agents 327-1, . . . , 327-N on the VCIs 306.In some embodiments, the data collection agents 328 can periodicallyforward the information and/or scores at set or configurable timeintervals. In one non-limiting example, the data collection agents 328can forward the information and/or scores to the configuration agents328 every few or tens of milliseconds (e.g., every 30 milliseconds,etc.). Embodiments are not so limited, however, and in some embodiments,the data collection agents 328 can forward the information and/or scoresto the configuration agents 327 in response to a detection that athreshold change has occurred in the information and/or scores since thelast information and/or scores were sent to the configuration agents327.

The configuration agents 327 can advertise or forward the informationand/or scores received from the data collection agents 328 to themicroservice scheduling agent 307. In some embodiments, theconfiguration agents 327 can advertise the information and/or scores tothe microservice scheduling agent 307 via an application programminginterface (API) call, or the configuration agents 327 can forward theinformation and/or scores to the microservice scheduling agent 307periodically or in response to receipt of the information and/or scoresfrom the data collection agents 328.

If, in the example shown in FIG. 3A, the cluster 305 includes twohypervisors 304-1 and 304-N and there are more long lived heavy flowsrunning inside the container(s)320 _(A)-1, . . . , 320 _(A)-N on the VCI306-1 than there are running on the container(s)320 _(B) on the VCI306-2, the score for the VCI 306-1 will be lower than the score for theVCI 306-2. In some embodiments, a weighting of other network states,such as pnic statistics or tunnel states associated with the hypervisors304-1, . . . , 304-N may also be factored into the score.

If a new container is to be created, the microservice scheduling agent307 can determine on which VCI 306 to schedule the container deploymentbased on resources available to the VCIs 306 in addition to the networkstate scores. By including the determined network state information inthe calculus performed by the microservice scheduling agent 307 inaddition to the resources available to the VCIs 306 when schedulingdeployment of new containers 320, performance of microservices and theapplications that depend on the microservices 322 can be improved incomparison to approaches in which only the resources available to theVCIs 306 are taken into account. In addition, because the network stateinformation and/or scores can be asynchronously sent by the datacollection agents 328, delays in network traffic may be furthermitigated in comparison to some approaches.

As shown in FIG. 3A, the cluster 305 (e.g., the VCC) can include aplurality of virtual computing instances (VCIs) 306 provisioned with apool of computing resources (e.g., processing resources 108 and/ormemory resources 110 illustrated in FIG. 1 , herein) and ultimatelyexecuted by hardware. In some embodiments, at least a first VCI (e.g.,the VCI 306-1) is deployed on a first hypervisor (e.g., the hypervisor304-1) of the VCC 305 and at least a second VCI (e.g., the VCI 306-2) isdeployed on a second hypervisor (e.g., the hypervisor 304-N) of the VCC305. The VCIs 306 can include containers 320.

In some embodiments, the VCC 305 can include a scheduling agent (e.g.,the microservice scheduling agent 307) that can be configured toreceive, from a first configuration agent 327-1 deployed on the firstVCI 306-1, a score corresponding to a network state of the firsthypervisor 304-1 and receive, from a second configuration agent 327-2deployed on the second VCI 306-2, a score corresponding to a networkstate of the second hypervisor 304-N. The scheduling agent 307 canfurther be configured to cause a container 320 to be deployed on atleast one of the first VCI 306-1 and the second VCI 306-2 based, atleast in part, on the score corresponding to a network state of thefirst hypervisor 304-1 and the score corresponding to a network state ofthe second hypervisor 304-N.

The scheduling agent 307 can be further configured to cause thecontainer to be deployed based, at least in part, on a determinationthat at least one network state for the first VCI 306-1 or the secondVCI 306-2 meets a network state criterion. As described above, thenetwork state criteria can include pnic statistics, tunnel statestatistics, network flow statistics, scores, etc. In some embodiments,the scheduling agent 307 is further configured to cause the container320 to be deployed on the first VCI 306-1 or the second VCI 306-2 based,at least in part, on a determination that a latency associated withexecution of a microservice 322 running on the first VCI 306-1 or thesecond VCI 306-2 has exceeded a threshold latency value. The latencyvalue can be determined by the scheduling agent 307, the schedulingsub-agents 326, the configuration agents 327, and/or the data collectionagent 328. The latency value may correspond to an amount of time amicroservice 322 takes to execute or may be based on an amount of timean application using the microservices 322 takes to execute.

In some embodiments, a first data collection agent 328-1 may be deployedon the first hypervisor 304-1. The first data collection agent 328-1 maybe configured to generate the first score based, at least in part, oncollected network state data and send the first score the to the firstconfiguration agent 237-1. A second data collection agent 328-N can bedeployed on the second hypervisor 304-N. The second data collectionagent 328-N can be configured to generate the second score based, atleast in part, on collected network state data and send the second scorethe to the first configuration agent.

The scheduling agent 307 can be configured to receive the first scorebased, at least in part, on a determination by the first configurationagent that the first score has changed greater than a threshold amountwithin a predetermined time period and/or receive the second scorebased, at least in part, on a determination by the second configurationagent that the second score has changed greater than a threshold amountwithin the predetermined time period. For example, if the first score orthe second score do not change during a predetermined threshold periodof time, the scheduling agent 307 may not receive updated scoreinformation corresponding to the first score and/or the second score.However, if the first score and/or the second score change more than athreshold amount during the predetermined period of time, the schedulingagent 307 may receive updated score information corresponding to thefirst score and/or the second score. This may reduce additional trafficwhen the network states aren't changing much while still allowing forupdated network state information (e.g., scores) to be updated by thescheduling agent 307 when the state of the network changes. In someembodiments, the scheduling agent 307 can be further configured toschedule a microservice 322 on the container 320 based, at least inpart, on the score corresponding to the network state of the firsthypervisor 304-1 and the score corresponding to the network state of thesecond hypervisor 304-N.

FIG. 3B is another diagram of a system including a scheduling agent 307,virtual computing instances 306, and hypervisors 304 for microservicescheduling according to the present disclosure. As shown in FIG. 3B, thesystem, which can be a virtual computing cluster (VCC) 305, includes amicroservice scheduling agent 307, a plurality of VCIs 306-1, . . . ,306-N, and a plurality of hypervisors 304-1, . . . , 304-N. Theplurality of VCIs 306 can include respective containers 320, which canrun respective microservices 322 (e.g., microservices 322 _(A)-1, . . ., 322 _(A)-N, 322 _(B), 322 _(M), etc.). In addition, the respectiveVCIs 306 can include respective scheduling sub-agents 326-1, . . . ,326-N and data collection agents 328-1, . . . , 328-N.

In some embodiments, the VCC 305 can include a scheduling agent 307 thatcan be configured to receive, from a first data collection agent 328-1deployed on the first VCI 306-1, network state information correspondingto the first VCI 306-1. The scheduling agent 307 can also receive, froma second data collection agent 328-2 deployed on the second VCI 306-1,network state information corresponding to the second VCI 306-1. Thescheduling agent 307 can be further configured to cause a container 320to be deployed on at least one of the first VCI 306-1 and the second VCI306-2 based, at least in part, on the network state informationcorresponding to the first VCI 306-1 and the network state informationcorresponding to the second VCI 306-2.

As described above, the network state information can include statisticssuch as pnic statistics, latency statistics, etc. In some embodiments,the network state information can include information corresponding todata traffic in the VCC 305. The data traffic can be classified based onthe size of flows corresponding to the data traffic. For example, datatraffic corresponding to small flows, which may be referred to as “miceflows” can include flows that are approximately 10 kilobytes in size orless, while data traffic corresponding to large flows, which may bereferred to as “elephant flows” can include flows that are approximately10 kilobytes in size or greater. In some embodiments, the datacollection agents 328 can analyze data traffic to determine a quantity(e.g., an number) of mice flows and a quantity of elephant flowsassociated with the VCIs 306. This information can then be used by thescheduling agent 307 to determine which VCIs to deploy containers 320 torun microservices 322 on.

Further, as described above in connection with FIG. 3A, the datacollection agents 328 can generate respective scores for the VCIs 306based on the network state information collected by the data collectionagents 328. These scores may be sent to the scheduling agent 307 and/orthe scheduling sub-agents 326 as part of microservice schedulingaccording to the disclosure. For example, the scheduling agent 307 canschedule microservice 322 deployment on containers 320 running on theVCIs 306 based on the respective generated scores for the network stateinformation.

FIG. 4A is a flow diagram representing a method 440 for microservicescheduling according to the present disclosure. At block 442, the method440 can include determining a network state for a first hypervisor in avirtual computing cluster (VCC). The VCC can be analogous to the cluster305 illustrated in FIGS. 3A and 3B, herein. In some embodiments,determining the network state for the first hypervisor and the secondhypervisor can include determining a tunnel state corresponding to atleast one of the first hypervisor and the second hypervisor. Embodimentsare not so limited, however, and determining the network state for thefirst hypervisor and/or the second hypervisor can include determiningvarious network state criteria described above in connection with FIGS.3A and 3B. In some embodiments, the container to run the microservicecan be deployed on the VCI based on the determined tunnel state for thefirst hypervisor and/or the second hypervisor.

At block 444, the method 440 can include determining a network state fora second hypervisor in the VCC. In some embodiments, determining thenetwork state for the second hypervisor can include determining a tunnelstate corresponding to the second hypervisor. Embodiments are not solimited, however, and determining the network state for the secondhypervisor can include determining various network state criteriadescribed above in connection with FIGS. 3A and 3B. In some embodiments,the container to run the microservice can be deployed on the VCI basedon the determined tunnel state for the first hypervisor and/or thesecond hypervisor.

At block 446, the method 440 can include deploying a container to run amicroservice on a virtual computing instance (VCI) deployed on the firsthypervisor or the second hypervisor based, at least in part, on thedetermined network state for the first hypervisor and the secondhypervisor. The network state can be determined as described above inconnection with FIGS. 3A and 3B. For example, the container can bedeployed on the VCI based on a determination that at least one of thenetwork states for the first hypervisor or the second hypervisor meets anetwork state criterion. In some embodiments, deploying the container onthe VCI can be based, at least in part, on the determined tunnel statefor the first hypervisor or the second hypervisor. In other embodiments,the container can be deployed on the VCI based, at least in part, on adetermination that a microservice ranking score generated by a datacollection agent running on the first hypervisor or the secondhypervisor is lower for the hypervisor on which the VCI is deployed.

Embodiments are not so limited, however, and the container can bedeployed on the VCI based, at least in part, on a determination that thecontainer is to be operational for less than a threshold period of time.For example, the container can be deployed on the VCI based on adetermination that the container is short lived (e.g., has amicroservice running thereon). In other embodiments, the container canbe deployed on the VCI based on a determination that a ratio of elephantflows to mice flows executed in the VCC falls within a predeterminedrange. For example, if a threshold amount of traffic on one of thehypervisors corresponds to elephant flows and the other hypervisor has alower amount of traffic corresponding to elephant flows associatedtherewith, the container can be deployed on a VCI that is deployed onthe hypervisor having the smaller percentage of elephant flowsassociated therewith.

In some embodiments, a latency associated with execution of anapplication corresponding to the microservice can be determined, and thecontainer can be deployed on the VCI based, at least in part, on adetermination that the latency associated with execution of theapplication has exceeded a threshold application latency value. Thelatency can refer to an amount of additional time it takes to run anapplication due to the network states and/or memory usage of the VCIsand/or hypervisors on which the VCIs are deployed. Latencies for eachmicroservice can, in some examples, be summed to yield an overalllatency associated with execution of the application.

Determining the network state can also include determining averagenetwork utilization associated with uplinks corresponding to the firsthypervisor and/or the second hypervisor, as described above. In someembodiments, the container can be deployed on the VCI based on theaverage network utilization associated with uplinks corresponding to thehypervisor on which the VCI is deployed.

In some embodiments, the method 440 can comprise generating a virtualcomputing cluster (VCC), such as the cluster 305 illustrated in FIGS. 3Aand 3B, herein. Once the VCC is generated, a data collection agent(e.g., the data collection agents 328 illustrated in FIGS. 3A and 3B,herein) can collect information corresponding to network states of theVCC. As described in connection with FIGS. 3A and 3B, the datacollection agent(s) can generate scores for the network states andforward the network state information and/or the scores to configurationagents (e.g., the configuration agents 327 illustrated in FIGS. 3A and3B), which can be deployed on VCIs (e.g., the VCIs 306 illustrated inFIGS. 3A and 3B) of the VCC. For example, the network state can bedetermined using a data collection agent running on the first hypervisoror the second hypervisor. In such examples, deploying the container onthe VCI can be further based on a determination that a microserviceranking score generated by a data collection agent running on the firsthypervisor or the second hypervisor is lower for the hypervisor on whichthe VCI is deployed.

The method 440 can further include advertising, by the configurationagents, the network state information and/or scores received from thedata collection agents to a microservice scheduling agent (e.g., themicroservice scheduling agent 307 illustrated in FIGS. 3A and 3B) of theVCC. The microservice scheduling agent can be configured to scheduledeployment of containers (e.g., the containers 320 illustrated in FIGS.3A and 3B) based on the network state information and/or the scoresreceived from the configuration agents.

FIG. 4B is another flow diagram representing a method 450 formicroservice scheduling according to the present disclosure. At block452, the method 450 can include determining a network state for a firstVCI. In some embodiments, determining the network state for the firstVCI can include determining a tunnel state corresponding to the firstVCI. Embodiments are not so limited, however, and determining thenetwork state for the first VCI can include determining various networkstate criteria described above in connection with FIGS. 3A and 3B.

At block 454, the method 450 can include determining a network state fora second VCI. In some embodiments, determining the network state for thesecond VCI can include determining a tunnel state corresponding to thesecond VCI. Embodiments are not so limited, however, and determining thenetwork state for the second VCI can include determining various networkstate criteria described above in connection with FIGS. 3A and 3B.

At block 456, the method 450 can include deploying a container to run amicroservice on the first VCI or the second VCI based, at least in part,on the determined network state for the first VCI and the second VCI.The network state can be determined as described above in connectionwith FIGS. 3A and 3B. For example, the container can be deployed on thefirst VCI or the second VCI based on a determination that at least oneof the network states for the hypervisor meets a network statecriterion. In some embodiments, deploying the container on the first VCIor the second VCI can be based, at least in part, on the determinedtunnel state for the first VCI or the second VCI. In other embodiments,the container can be deployed on the first VCI or the second VCI based,at least in part, on a determination that a microservice ranking scoregenerated by a data collection agent communicatively coupled to thefirst VCI and the second VCI is lower for the second VCI than for thefirst VCI.

Embodiments are not so limited, however, and the container can bedeployed on the first VCI or the second VCI based, at least in part, ona determination that the container is to be operational for less than athreshold period of time. For example, the container can be deployed onthe first VCI or the second VCI based on a determination that thecontainer is short lived (e.g., has a microservice running thereon). Insome embodiments, a latency associated with execution of an applicationcorresponding to the microservice can be determined, and the containercan be deployed on the first VCI or the second VCI based, at least inpart, on a determination that the latency associated with execution ofthe application has exceeded a threshold application latency value. Thelatency can refer to an amount of additional time it takes to run anapplication due to the network states and/or memory usage of the VCIsand/or hypervisors on which the VCIs are deployed. Latencies for eachmicroservice can, in some examples, be summed to yield an overalllatency associated with execution of the application.

Determining the network state can also include determining averagenetwork utilization associated with uplinks corresponding to ahypervisor on which the first VCI or the second VCI is deployed, asdescribed above. In some embodiments, the container can be deployed onthe first VCI or the second VCI based on the average network utilizationassociated with uplinks corresponding to the hypervisor on which thefirst VCI or the second VCI is deployed.

In some embodiments, the method can include determining the networkstates for the first VCI and the second VCI using a data collectionagent such as the data collection agents 328 illustrated and describedabove in connection with FIGS. 3A and 3B. For example, the informationcorresponding to the network states for the first VCI and the second VCIcan be collected by a data collection agent deployed on a hypervisor, asshown in FIG. 3A, or the information corresponding to the network statesfor the first VCI and the second VCI can be collected by a datacollection agent deployed on the VCIs, as shown in FIG. 3B.

The method can further include determining a network state for a firstvirtual computing instance (VCI) on which a container executing themicroservice is deployed. In some embodiments, determining the networkstate for the first VCI can include determining a tunnel statecorresponding to the first VCI. Embodiments are not so limited, however,and determining the network state for the first VCI can includedetermining various network state criteria described above in connectionwith FIGS. 3A and 3B.

The method can include determining a network state for a second VCI. Insome embodiments, determining the network state for the second VCI caninclude determining a tunnel state corresponding to the second VCI.Embodiments are not so limited, however, and determining the networkstate for the second VCI can include determining various network statecriteria described above in connection with FIGS. 3A and 3B.

The method can include re-deploying the container on the second VCIbased, at least in part, on the determined network state for the firstVCI and the second VCI. The network state can be determined as describedabove in connection with FIGS. 3A and 3B. For example, the container canbe deployed (or re-deployed) on first VCI or the second VCI based on adetermination that at least one of the network states for the hypervisormeets a network state criterion. In some embodiments, deploying (orre-deploying) the container on the first VCI or the second VCI can bebased, at least in part, on the determined tunnel state for the firstVCI or the second VCI. In other embodiments, the container can bedeployed (or re-deployed) on the first VCI or the second VCI based, atleast in part, on a determination that a microservice ranking scoregenerated by a data collection agent communicatively coupled to thefirst VCI and the second VCI is lower for the second VCI than for thefirst VCI.

Embodiments are not so limited, however, and the container can bedeployed (or re-deployed) on the first VCI or the second VCI based, atleast in part, on a determination that the container is to beoperational for less than a threshold period of time. For example, thecontainer can be deployed (or re-deployed) on the first VCI or thesecond VCI based on a determination that the container is short lived(e.g., has a microservice running thereon). In some embodiments, alatency associated with execution of an application corresponding to themicroservice can be determined, and the container can be deployed (orre-deployed) on the first VCI or the second VCI based, at least in part,on a determination that the latency associated with execution of theapplication has exceeded a threshold application latency value. Thelatency can refer to an amount of additional time it takes to run anapplication due to the network states and/or memory usage of the VCIsand/or hypervisors on which the VCIs are deployed. Latencies for eachmicroservice can, in some examples, be summed to yield an overalllatency associated with execution of the application.

Determining the network state can also include determining averagenetwork utilization associated with uplinks corresponding to ahypervisor on which the first VCI or the second VCI is deployed, asdescribed above. In some embodiments, the container can be deployed onthe first VCI or the second VCI based on the average network utilizationassociated with uplinks corresponding to the hypervisor on which thefirst VCI or the second VCI is deployed.

In some embodiments, the method can comprise generating a virtualcomputing cluster (VCC), such as the cluster 305 illustrated in FIGS. 3Aand 3B, herein. Once the VCC is generated, a data collection agent(e.g., the data collection agents 328 illustrated in FIGS. 3A and 3B,herein) can collect information corresponding to network states of theVCC. As described in connection with FIGS. 3A and 3B, the datacollection agent(s) can generate scores for the network states andforward the network state information and/or the scores to configurationagents (e.g., the configuration agents 327 illustrated in FIGS. 3A and3B), which can be deployed on VCIs (e.g., the VCIs 306 illustrated inFIGS. 3A and 3B) of the VCC.

The method can further include advertising, by the configuration agents,the network state information and/or scores received from the datacollection agents to a microservice scheduling agent (e.g., themicroservice scheduling agent 307 illustrated in FIGS. 3A and 3B) of theVCC. The microservice scheduling agent can be configured to scheduledeployment of containers (e.g., the containers 320 illustrated in FIGS.3A and 3B) based on the network state information and/or the scoresreceived from the configuration agents.

FIG. 5 is a diagram of an apparatus for microservice schedulingaccording to the present disclosure. The apparatus 514 can include adatabase 515, a subsystem 518, and/or a number of engines, for examplemicroservice scheduling engine 519, and can be in communication with thedatabase 515 via a communication link. The apparatus 514 can includeadditional or fewer engines than illustrated to perform the variousfunctions described herein. The apparatus 514 can represent programinstructions and/or hardware of a machine (e.g., machine 630 asreferenced in FIG. 6 , etc.). As used herein, an “engine” can includeprogram instructions and/or hardware, but at least includes hardware.Hardware is a physical component of a machine that enables it to performa function. Examples of hardware can include a processing resource, amemory resource, a logic gate, etc. In some embodiments, the apparatus514 can be analogous to the microservice scheduling agent 114illustrated and described in connection with FIG. 1 , herein.

The engines (e.g., 519) can include a combination of hardware andprogram instructions that are configured to perform a number offunctions described herein. The program instructions (e.g., software,firmware, etc.) can be stored in a memory resource (e.g.,machine-readable medium) as well as hard-wired program (e.g., logic).Hard-wired program instructions (e.g., logic) can be considered as bothprogram instructions and hardware.

In some embodiments, the microservice scheduling engine 519 can includea combination of hardware and program instructions that can beconfigured to determine a network state for a first virtual computinginstance (VCI) and a second VCI on which a first container and a secondcontainer are respectively deployed. The microservice scheduling engine519 can cause the first container to be deployed (or re-deployed) on thefirst VCI or the second VCI based, at least in part, on a determinationthat at least one of the network states for the first or second VCImeets a network state criterion. The network state criterion can includeone or more of the network state criteria described above in connectionwith FIGS. 3A and 3B.

The microservice scheduling engine 519 can further include a combinationof hardware and program instructions that can be configured to determinea network state for a hypervisor on which the first VCI or the secondVCI is deployed and cause the first container to be deployed on thefirst VCI or the second VCI based, at least in part, on a determinationthat at least one of the network states for the hypervisor meets ahypervisor state criterion.

The microservice scheduling engine 519 can further include a combinationof hardware and program instructions that can be configured to cause thefirst container to be deployed on the first VCI or the second VCI based,at least in part, on a determined average network utilization associatedwith an uplink corresponding to the hypervisor. In some embodiments, themicroservice scheduling engine 519 can be configured to re-allocatecomputing resources to at least one of the first container and thesecond container based, at least in part, on the determination that theat least one of the network states for the first or second VCI meets thenetwork state criterion, a latency associated with execution of the flowrunning on the first container has exceeded a threshold, or acombination thereof.

The microservice scheduling engine 519 can further include a combinationof hardware and program instructions that can be configured to determinea tunnel state for the first VCI or the second VCI as part of thedetermination of the network state for the first VCI or the second VCI.

The microservice scheduling engine 519 can further include a combinationof hardware and program instructions that can be configured to cause thefirst container to be deployed on the first VCI or the second VCI based,at least in part, on a determination that the first container is to beoperational for less than a threshold period of time.

The microservice scheduling engine 519 can further include a combinationof hardware and program instructions that can be configured to cause thefirst container to be deployed on first VCI or the second VCI based, atleast in part, on a determined average network utilization associatedwith a virtual network interface card corresponding to a hypervisor onwhich the first VCI and/or the second VCI is deployed.

FIG. 6 is a diagram of a machine for packet generation and injectionaccording to the present disclosure. The machine 630 can utilizesoftware, hardware, firmware, and/or logic to perform a number offunctions. The machine 630 can be a combination of hardware and programinstructions configured to perform a number of functions (e.g.,actions). The hardware, for example, can include a number of processingresources 608 and a number of memory resources 610, such as amachine-readable medium (MRM) or other memory resources 610. The memoryresources 610 can be internal and/or external to the machine 630 (e.g.,the machine 630 can include internal memory resources and have access toexternal memory resources). In some embodiments, the machine 630 can bea VCI, for example, the machine 630 can be a server. The programinstructions (e.g., machine-readable instructions (MRI)) can includeinstructions stored on the MRM to implement a particular function (e.g.,an action such as packet generation and/or injection). The set of MRIcan be executable by one or more of the processing resources 608. Thememory resources 610 can be coupled to the machine 630 in a wired and/orwireless manner. For example, the memory resources 610 can be aninternal memory, a portable memory, a portable disk, and/or a memoryassociated with another resource, e.g., enabling MRI to be transferredand/or executed across a network such as the Internet. As used herein, a“module” can include program instructions and/or hardware, but at leastincludes program instructions.

Memory resources 610 can be non-transitory and can include volatileand/or non-volatile memory. Volatile memory can include memory thatdepends upon power to store information, such as various types ofdynamic random-access memory (DRAM) among others. Non-volatile memorycan include memory that does not depend upon power to store information.Examples of non-volatile memory can include solid state media such asflash memory, electrically erasable programmable read-only memory(EEPROM), phase change random access memory (PCRAM), magnetic memory,optical memory, and/or a solid-state drive (SSD), etc., as well as othertypes of machine-readable media.

The processing resources 608 can be coupled to the memory resources 610via a communication path 631. The communication path 631 can be local orremote to the machine 630. Examples of a local communication path 631can include an electronic bus internal to a machine, where the memoryresources 610 are in communication with the processing resources 608 viathe electronic bus. Examples of such electronic buses can includeIndustry Standard Architecture (ISA), Peripheral Component Interconnect(PCI), Advanced Technology Attachment (ATA), Small Computer SystemInterface (SCSI), Universal Serial Bus (USB), among other types ofelectronic buses and variants thereof. The communication path 631 can besuch that the memory resources 610 are remote from the processingresources 608, such as in a network connection between the memoryresources 610 and the processing resources 608. That is, thecommunication path 631 can be a network connection. Examples of such anetwork connection can include a local area network (LAN), wide areanetwork (WAN), personal area network (PAN), and the Internet, amongothers.

As shown in FIG. 6 , the MRI stored in the memory resources 610 can besegmented into a number of modules, e.g., 633, that when executed by theprocessing resource(s) 608, can perform a number of functions. As usedherein a module includes a set of instructions included to perform aparticular task or action. The module(s) 633 can be sub-modules of othermodules. Examples are not limited to the specific module(s) 633illustrated in FIG. 6 .

The module(s) 633 can include program instructions and/or a combinationof hardware and program instructions that, when executed by a processingresource 608, can function as a corresponding engine as described withrespect to FIG. 5 . For example, the microservice scheduling module 633can include program instructions and/or a combination of hardware andprogram instructions that, when executed by a processing resource 608,can function as the microservice scheduling engine 519.

Although specific embodiments have been described above, theseembodiments are not intended to limit the scope of the presentdisclosure, even where only a single embodiment is described withrespect to a particular feature. Examples of features provided in thedisclosure are intended to be illustrative rather than restrictiveunless stated otherwise. The above description is intended to cover suchalternatives, modifications, and equivalents as would be apparent to aperson skilled in the art having the benefit of this disclosure.

The scope of the present disclosure includes any feature or combinationof features disclosed herein (either explicitly or implicitly), or anygeneralization thereof, whether or not it mitigates any or all of theproblems addressed herein. Various advantages of the present disclosurehave been described herein, but embodiments may provide some, all, ornone of such advantages, or may provide other advantages.

In the foregoing Detailed Description, some features are groupedtogether in a single embodiment for the purpose of streamlining thedisclosure. This method of disclosure is not to be interpreted asreflecting an intention that the disclosed embodiments of the presentdisclosure have to use more features than are expressly recited in eachclaim. Rather, as the following claims reflect, inventive subject matterlies in less than all features of a single disclosed embodiment. Thus,the following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separateembodiment.

What is claimed is:
 1. A method for containerized workload scheduling,comprising: determining, at a scheduling agent of a virtual computingcluster (VCC), a network state for: a first hypervisor in the VCC; and asecond hypervisor in the VCC; wherein the network state is based atleast in part on information derived from network traffic of the VCC,wherein the determining the network state for the first hypervisor andthe second hypervisor further comprises: determining average networkutilization associated with uplinks corresponding to the firsthypervisor and the second hypervisor; and deploying, by the schedulingagent, a container to run a containerized workload on a virtualcomputing instance (VCI) deployed on the first hypervisor or the secondhypervisor based, at least in part, on the determined network state forthe first hypervisor and the second hypervisor and on the averagenetwork utilization associated with uplinks corresponding to the firsthypervisor and the second hypervisor.
 2. The method of claim 1, furthercomprising determining a network state for the first hypervisor and thesecond hypervisor using a data collection agent running on the firsthypervisor or the second hypervisor.
 3. The method of claim 1, whereindetermining the network state for the first hypervisor and the secondhypervisor further comprises determining a tunnel state corresponding toat least one of the first hypervisor and the second hypervisor, andwherein deploying the container on the VCI is further based, at least inpart, on the determined tunnel state for the first hypervisor and thesecond hypervisor.
 4. The method of claim 1, wherein deploying thecontainer on the VCI is further based, at least in part, on adetermination that a containerized workload ranking score generated by adata collection agent running on the first hypervisor or the secondhypervisor is lower for the hypervisor on which the VCI is deployed. 5.The method of claim 1, wherein deploying the container on the VCI isfurther based, at least in part, on a determination that a ratio ofelephant flows to mice flows executed in the VCC falls within apredetermined range.
 6. The method of claim 1, further comprisingdetermining a latency associated with execution of an applicationcorresponding to the containerized workload, wherein deploying thecontainer on the VCI is further based, at least in part, on adetermination that the latency associated with execution of theapplication has exceeded a threshold application latency value.
 7. Themethod of claim 1, wherein determining the network state for the firsthypervisor and the second hypervisor further comprises determininginterference information corresponding to non-containerized resourcesrunning in the VCC, and wherein deploying the container on the first VCIor the second VCI is further based, at least in part, on the determinedinterference information.
 8. A method for containerized workloadscheduling, comprising: determining, at a scheduling agent of a virtualcomputing cluster (VCC), a network state for: a first virtual computinginstance (VCI) of the VCC comprising a first scheduling sub-agent; and asecond VCI of the VCC comprising a second scheduling sub-agent; whereinthe first scheduling sub-agent and the second scheduling sub-agentcommunicate network state information corresponding to the first VCI andthe second VCI to the scheduling agent for use in determining thenetwork state, wherein the network state is based at least in part oninformation derived from network traffic of the VCC; and deploying, bythe scheduling agent, a container to run a containerized workload on thefirst VCI or the second VCI based, at least in part, on the determinednetwork state for the first VCI and the second VCI.
 9. The method ofclaim 8, wherein determining the network state for the first VCI and thesecond VCI further comprises determining a tunnel state corresponding toat least one of the first VCI and the second VCI, and wherein deployingthe container on the first VCI or the second VCI is further based, atleast in part, on the determined tunnel state for the first VCI or thesecond VCI.
 10. The method of claim 8, wherein determining the networkstate for the first VCI and the second VCI further comprises determininginterference information corresponding to non-containerized resourcesrunning in a virtual computing cluster in which the first VCI and thesecond VCI are deployed, and wherein deploying the container on thefirst VCI or the second VCI is further based, at least in part, on thedetermined interference information.
 11. The method of claim 8, whereindetermining the network state for the first VCI and the second VCIfurther comprises determining the network state using a data collectionagent deployed on at least one of the first VCI and the second VCI. 12.The method of claim 8, wherein determining the network state for thefirst VCI and the second VCI further comprises determining the networkstate using a data collection agent deployed on at least one of a firsthypervisor on which the first VCI is deployed and a second hypervisor onwhich the second VCI is deployed.
 13. The method of claim 8, whereindeploying the container on the first VCI or the second VCI is furtherbased, at least in part, on a determination that the container is to beoperational for less than a threshold period of time.
 14. The method ofclaim 8, wherein determining the network state for the first VCI and thesecond VCI further comprises determining average network utilizationassociated with uplinks corresponding to a hypervisor on which the firstVCI or the second VCI is deployed, and wherein deploying the containeron the first VCI or the second VCI is further based, at least in part,on the average network utilization associated with an uplinkcorresponding to the hypervisor.
 15. An apparatus for containerizedworkload scheduling, comprising: a memory device; and a hardwareprocessor configured to execute: a containerized workload schedulingagent of a virtual computing cluster (VCC) and provisioned withprocessing resources and configured to execute instructions to:determine a network state for a first virtual computing instance (VCI)of the VCC comprising a first scheduling sub-agent and a second VCI ofthe VCC comprising a second scheduling sub-agent, wherein the firstscheduling sub-agent and the second scheduling sub-agent communicatenetwork state information corresponding to the first VCI and the secondVCI to the scheduling agent for use in determining the network state,wherein the network state is based at least in part on informationderived from network traffic of the VCC; and cause container to run acontainerized workload to be deployed on the first VCI or the second VCIbased, at least in part, on a determination that at least one of thenetwork states for the first VCI or the second VCI meets a network statecriterion.
 16. The apparatus of claim 15, wherein the containerizedworkload scheduling agent is further configured to: determine a networkstate for a hypervisor on which the first VCI or the second VCI isdeployed; and cause the container to be deployed on the first VCI or thesecond VCI based, at least in part, on a determination that at least oneof the network states for the hypervisor meets a hypervisor statecriterion.
 17. The apparatus of claim 16, wherein the containerizedworkload scheduling agent is further configured to cause the containerto be deployed on the first VCI or the second VCI based, at least inpart, on a determined average network utilization associated with anuplink corresponding to the hypervisor.
 18. The apparatus of claim 15,wherein the containerized workload scheduling agent is furtherconfigured to re-allocate computing resources to at least one of thefirst container and the second container based, at least in part, on thedetermination that the at least one of the network states for the firstor second VCI meets the network state criterion, a latency associatedwith execution of the flow running on the first container has exceeded athreshold, or a combination thereof.
 19. The apparatus of claim 15,wherein the containerized workload scheduling agent is furtherconfigured to determine a tunnel state for the first VCI or the secondVCI as part of the determination of the network state for the first VCIor the second VCI.
 20. The apparatus of claim 15, wherein thecontainerized workload scheduling agent is further configured to causethe container to be deployed on the first VCI or the second VCI based,at least in part, on a determination that the container is to beoperational for less than a threshold period of time.
 21. The apparatusof claim 15, wherein the containerized workload scheduling agent isfurther configured to cause the container to be deployed on the firstVCI or the second VCI based, at least in part, on a determined averagenetwork utilization associated with a virtual network interface cardcorresponding to a hypervisor on which the first VCI or the second VCIis deployed.
 22. A system for containerized workload scheduling,comprising: a memory device; and a hardware processor configured toexecute: a virtual computing cluster (VCC) comprising a plurality ofvirtual computing instances (VCIs) provisioned with a pool of computingresources and ultimately executed by hardware, wherein at least a firstVCI is deployed on a first hypervisor of the VCC and wherein at least asecond VCI is deployed on a second hypervisor of the VCC; and ascheduling agent deployed on the VCC and configured to: receive, from afirst configuration agent deployed on the first VCI, a scorecorresponding to a network state of the first hypervisor, wherein thescore corresponding to a network state of the first hypervisor is basedat least in part on information derived from network traffic of the VCC;and receive, from a second configuration agent deployed on the secondVCI, a score corresponding to a network state of the second hypervisor,wherein the score corresponding to a network state of the secondhypervisor is based at least in part on information derived from thenetwork traffic of the VCC; and cause a container to be deployed on atleast one of the first VCI and the second VCI based, at least in part,on the score corresponding to the network state of the first hypervisorand the score corresponding to the network state of the secondhypervisor.
 23. The system of claim 22, wherein the scheduling agent isfurther configured to cause the container to be deployed based, at leastin part, on a determination that at least one network state for thefirst VCI or the second VCI meets a network state criterion.
 24. Thesystem of claim 22, wherein the scheduling agent is further configuredto cause the container to be deployed on the first VCI or the second VCIbased, at least in part, on a determination that a latency associatedwith execution of a containerized workload running on the first VCI orthe second VCI has exceeded a threshold latency value.
 25. The system ofclaim 22, further comprising a first data collection agent deployed onthe first hypervisor and configured to: generate the first score based,at least in part, on collected network state data; and send the firstscore the to the first configuration agent; and a second data collectionagent deployed on the second hypervisor and configured to: generate thesecond score based, at least in part, on collected network state data;and send the second score the to the first configuration agent.
 26. Thesystem of claim 22, wherein the scheduling agent is configured to:receive the first score based, at least in part, on a determination bythe first configuration agent that the first score has changed greaterthan a threshold amount within a predetermined time period; and receivethe second score based, at least in part, on a determination by thesecond configuration agent that the second score has changed greaterthan a threshold amount within the predetermined time period.
 27. Thesystem of claim 22, wherein the scheduling agent is further configuredto schedule a containerized workload on the container based, at least inpart, on the score corresponding to the network state of the firsthypervisor and the score corresponding to a network state of the secondhypervisor.
 28. The system of claim 22, further comprising a first datacollection agent deployed on the first hypervisor and configured to:generate the first score based, at least in part, on interferencestatistics corresponding to non-containerized workloads running on thefirst VCI; and send the first score the to the first configurationagent; and a second data collection agent deployed on the secondhypervisor and configured to: generate the second score based, at leastin part, on interference statistics corresponding to non-containerizedworkloads running on the second VCI; and send the second score the tothe first configuration agent.
 29. The system of claim 22, wherein thescheduling agent is further configured to: determine interferenceinformation corresponding to non-containerized resources running in theVCC; and cause a container to be deployed on at least one of the firstVCI and the second VCI based, at least in part, on the determinedinterference information.
 30. A system for containerized workloadscheduling, comprising: a memory device; and a hardware processorconfigured to execute: a virtual computing cluster (VCC) comprising aplurality of virtual computing instances (VCIs) provisioned with a poolof computing resources and ultimately executed by hardware, wherein atleast a first VCI is deployed on a first hypervisor of the VCC andwherein at least a second VCI is deployed on a second hypervisor of theVCC; and a scheduling agent deployed on the VCC and configured to:receive, from a first data collection agent deployed on the first VCI,network state information corresponding to the first VCI, wherein thenetwork state information corresponding to the first VCI is based atleast in part on information derived from network traffic of the VCC;and receive, from a second data collection agent deployed on the secondVCI, network state information corresponding to the second VCI, whereinthe network state information corresponding to the second VCI is basedat least in part on information derived from the network traffic of theVCC; and cause a container to be deployed on at least one of the firstVCI and the second VCI based, at least in part, on the network stateinformation corresponding to the first VCI and the network stateinformation corresponding to the second VCI.
 31. The system of claim 30,wherein the scheduling agent is further configured to cause thecontainer to be deployed on the first VCI or the second VCI based, atleast in part, on network traffic behavior associated with the first VCIand the second VCI.
 32. The system of claim 30, wherein the first datacollection agent is configured to generate a first score based, at leastin part on the network state information corresponding to the first VCI,and wherein the first data collection agent is configured to generate asecond score based, at least in part on the network state informationcorresponding to the second VCI.
 33. The system of claim 32, wherein thescheduling agent is configured to: receive the first score; receive thesecond score; and cause the container to be deployed on at least one ofthe first VCI and the second VCI based, at least in part, on the firstscore and the second score.